Nonequilibrium Thermodynamics of Feedback Control 
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We establish a general theory of feedback control on classical stochastic thermodynamic systems, 
and generalize nonequilibrium equalities such as the fluctuation theorem and the Jarzynski equality 
in the presence of feedback control with multiple measurements. Our results are generalizations 
of the previous relevant works to the situations with general measurements and multi-heat baths. 
The obtained equalities involve additional terms that characterize the information obtained by 
measurements or the efficacy of feedback control. A generalized Szilard engine and a feedback- 
controlled ratchet are shown to satisfy the derived equalities. 
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I. INTRODUCTION 

Since the mid-twentieth century, feedback control has 
played crucial roles in science and engineering [l|, 
Here, "feedback" means that a control protocol depends 
on measurement outcomes obtained from the controlled 
system. Recently, feedback control has become increas- 
ingly important in terms of nonequilibrium physics, due 
to at least the following two reasons. 

First of all, stochastic aspects of thermodynamics 0- 

have become important due to recent theoretical and 
experimental developments. Theoretically, a number of 
nonequilibrium equalities such as the fluctuation theo- 
rem and the Jarzynski equality [Tol - l39| have recently 
been found. On the other hand, experimental tech- 
niques have been developed to manipulate and observe 
small thermodynamic systems such as macromolecules 
and colloidal particles, and several nonequilibrium equal- 
ities have been experimentally verified [40- 50] . Moreover, 
artificial [sH - ls^ ] and biological [S^ molecular machines 
have been investigated. In these contexts, feedback con- 
trol is useful to realize intended dynamical properties of 
small thermodynamic systems, and it has become a topic 
of active research [56l - l77| . 

Secondly, feedback control sheds light on the founda- 
tions of thermodynamics and statistical mechanics con- 
cerning "Maxwell's demon" [TSl - lssj . In fact. Maxwell's 
demon performs measurement and feedback control on 
thermodynamic systems. Recently, Maxwell's demon has 
attracted renewed interest 84i-i98l | from the standpoints 
of modern information theory and statistical mechanics. 

A quintessential model of Maxwell's demon is a single- 
particle heat engine proposed by L. Szilard in 1929 80] . 
During the thermodynamic cycle of the Szilard engine, 
the demon obtains 1 bit (= In 2 nat) of information by 
a measurement, performs feedback control, and extracts 
fcBrin2 of positive work from a single heat bath. After 
numerous arguments on the consistency between the de- 



mon and the second law of thermodynamics, it is now un- 
derstood that the work needed for the demon (or equiv- 
alently the feedback controller) during the measurement 
and information erasure compensates for the work that 
can be extracted by the demon [9^ . Therefore, we cannot 
extract a net positive work from the total system of the 
engine and the demon in an isothermal cycle, and there- 
fore the presence of the demon does not contradict the 
second law of thermodynamics. Nevertheless, kBT\n2 
of work extracted by the demon can be still useful. By 
using feedback control, we can increase the system's free 
energy without injecting any energy (work) to it. We 
stress that, without feedback control, we need the di- 
rect energy input into the system in order to increase its 
free energy due to the second law of thermodynamics. 
Feedback control may be regarded as a powerful tool to 
control thermodynamic systems. Since the crucial quan- 
tity is the information that is obtained to be used for 
feedback control, we may regard the Szilard-type heat 
engine as "information heat engine." Recently, such an 
information heat engine was realized experimentally by 
using a colloidal particle [z3] ■ 

In this paper, we formulate a general theory of feedback 
control on stochastic thermodynamic systems. In partic- 
ular, we extend recent theoretical results on the gener- 
alizations of the fluctuation theorem and the Jarzynski 
equality (63j to the situations in which the measurement 
and feedback control are non-Markovian and there are 
multi-heat baths. Our results serve as the fundamental 
building blocks of information heat engines. 

This paper is organized as follows. 

In Sec. II, we briefly review the framework of stochastic 
thermodynamics in a general setup. We discuss classical 
stochastic systems that are in general non-Markovian and 
in contact with multi-heat baths. We discuss the concept 
of entropy production and the detailed fluctuation the- 
orem as our starting point. Because they are general 
properties of nonequilibrium systems, our formulations 
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and results in the following sections are not restricted to 
Langevin systems but applicable to any classical stochas- 
tic systems that satisfy the detailed fluctuation theorem. 

In Sec. Ill, we formulate measurements on thermody- 
namic systems. We discuss multi-measurements includ- 
ing continuous measurements, and investigate the prop- 
erties of the mutual information obtained by the mea- 
surements. In particular, we introduce the two kinds of 
mutual information / and /c , which will be shown to play 
key roles in the discussion of feedback control. 

In Sec. IV, we discuss feedback control on Markov and 
non-Markov processes, and investigate feedback control 
in terms of probability theory, where the causality of the 
measurement and feedback play a crucial role. 

In Sec. V, we derive the main results of this paper. 
We generalize the nonequilibrium equalities to situations 
in which the system is subject to feedback control. In 
particular, we derive two types of generalizations of the 
fluctuation theorem and the Jarzynski equality. One in- 
volves a term concerning the mutual information, and 
the other involves a term of feedback efficacy. As corol- 
laries, we derive the generalizations of the second law of 
thermodynamics and a fluctuation-dissipation relation. 

In Sec. VI, we illustrate our general results by two ex- 
amples: a generalized Szilard engine with measurement 
errors and a feedback-controlled ratchet [s^ [H, [63] . We 
discuss the former analytically and the latter numerically. 

In Sec. VII, we conclude this paper. 

In Appendix A, we discuss the physical meaning of 
entropy production to elucidate the physical contents of 
our results in two typical situations. 



II. REVIEW OF STOCHASTIC 
THERMODYNAMICS 

In this section, we briefly review thermodynamics of 
classical stochastic systems and introduce notations that 
will be used later. 



Control protocol \{t) can also be discretized. Let A„ 
be the value of A between i„ = nAt and tn+i = {n+l)At, 
where it is assumed to be constant during this time in- 
terval (see FIG. 1). We denote the trajectory of A from 
time to i„ as A„ :— (Aq, Ai, • • • , An,_i). Let Aint be the 
value of parameter A before time 0, which is not neces- 
sarily equal to Ao because we can switch the value of the 
parameter at time 0. We also denote the value of A after 
time tif := T as Afin, which is not necessarily equal to 
Aat, either (see also FIG. 1). 

Let P„[a:n,] be the probability distribution of x at 
time tn- In particular, Po[2^o] is the initial distribution 
of X. The initial distribution can be chosen as a sta- 
tionary distribution under external parameters Aint, as 
-Ps[a;o|Aint], which means Po[xo] = f'siaJolAmt]- We note 
that Ps[a;o|Amt] is not necessarily a canonical distribution; 
it can be a nonequilibrium stationary distribution. Due 
to the causality, Xn+i is determined by Xn through the 
transition probability P[xn+i\Xn, A„], which depends on 
the external parameters at time t„ (i.e., A„). We note 
that P[xn+i\Xn, Xn] represents the the probability of re- 
alizing Xn+i at time tn+i under the condition that the 
trajectory of x up to time i„ is given by A„. If the dy- 
namics is Markovian, P[xn+i\Xn, Xn] can be replaced by 

P[Xn+i\Xn, Xn]- 



X 



/[ ^2 ^3 ^4 t 



A. Dynamics 

We consider a classical stochastic system S that is in 
contact with heat baths Bi, B2, B„ at respective 
temperatures Ti = {kB(3i)'^, T2 = (fcB;02)~^, •■•,?"„ = 
(fcB/3n)~^- Let X be the phase-space point of system S 
and A be a set of external parameters such as the volume 
of a gas or the frequency of an optical tweezers. We 
control the system from time to r with control protocol 
X{t). Let x{t) be a trajectory of the system. 

To formulate the stochastic dynamics, we discretize the 
time interval [0, r] by dividing it into N small intervals 
with width At := t/N. The original continuous-time 
dynamics is recovered by taking the limit of A'^ — >■ 00 or 
equivalently Ai — > 0. Let t = nAt and x„ := x{nAt). We 
refer to "time t" as "time t„ := nAt." Then, trajectory 
{a;(t')}t'6[o,t] corresponds to A„ := (zq, xi, • • • , a;„). 



FIG. 1: Discretization of control protocol \{t). 
The probability of trajectory A„ is then given by 

n-l 

P[Xn]K] = n P[xk+l]XkAk]PM =■■ P[Xn], (1) 
fe=0 

where we write P[A„|A„] just as P[A„] for simplicity. 
We note that 

P[Xn]xo,Kn] = n P[xk+i]Xk,Xk] =: P[A„|xo] (2) 

is the probability of trajectory X„ under the condition 
that the initial state is xq and the control protocol is A„. 
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Let A be an arbitrary physical quantity that can de- 
pend on the trajectory Xf^ and protocol Ajy. The en- 
semble average of this quantity is given by 



(A) = / dXNP[XN\AN]A[XN,AN], 



where dXjy := n^=o ^-^ 



(3) 



where the conditional probability under initial is given 

by 



(7) 



fe=0 



We note that dXlj = dX^ holds, because dxn = dx* . 



B. Backward Control 

Before proceeding to the nonequilibrium equalities, we 
consider the stochastic dynamics with a backward con- 
trol protocol. The backward control protocol means the 
time-reversal of protocol An, which is formulated as fol- 
lows. Let A* be the time-reversal of A; for example, if A 
is a magnetic field, then A* = —A. The time-reversed 
protocol of A(t) is then given by X^{t) :— A*(t — t). 
The backward protocol can be discretized as Ajj := 



We define Xl 



A 



N-r 



{X*N^l, X*^_2, ■ ■ • , X*j^_j^_^). 
\nt '^fin' ^fin '^hit ■ 

We consider the probability of realizing trajectory x'(t) 
of the system with a backward control protocol. We de- 
fine x'n := x'{nAt) and X^ := {xq,x'i, ■ ■ ■ ,x'j^). We de- 
note as P(|[a;o] the initial distribution of the backward 
processes. We stress that ^'^[a^o] is not necessarily equal 
to the final distribution of the forward experiments. In 
fact, we can prepare a new state for the system to perform 
the backward experiments after the forward experiments. 
The probability distribution of trajectory X'^ with back- 
ward protocol is given by 

N-l 

P{XM] = n PWk\XLxl]P^Wo] pHx'n], (4) 

fe=0 

where we write P[Xj^|A]y] as P^[X^] for simplicity. Cor- 
respondingly, 

N-l 

P[X'M,^U = n Pi^'klXL^i] =■■ P^X'nWo]- (5) 

In special cases, the backward trajectory X'j^ is equal 
to the time-reversal of the forward trajectory X^. Let x* 
be the time-reversal of phase-space point x. For example, 
if a; = (r,p) with r and p being the position and the 
momentum respectively, we have x* := {r,^p). The 
time-reversal of trajectory X„ is then given by X^ := 
With notation x^ := x%_^, we 

a;}j). By substituting — xl^ to 



write Xl = (xj, , ■ 

Eqs. (m and ([S]), we obtain the probability of realizing a 
backward trajectory under the backward protocol as 



N-l 



(6) 



C. Nonequilibrium Equalities 

We now discuss nonequilibrium equalities. Let 
Qi[XN , Xn] be the heat that is absorbed by the sys- 
tem from the ith heat bath satisfying Qi[XN, An] = 

-Qi[xl^,A''jy]. We write Q^[XN,XN] simply as Qi[XN] 
for simplicity. It has been established that the following 
equality is satisfied for stochastic thermodynamic sys- 
tems [i3,[ii|,[il[23: 



p'ixi 



Nl^a 



P[XN\xa] 



= exp 



l3iQi[XN] 



(8) 



which is referred to as the detailed fiuctuation theorem 
(or the transient fluctuation theorem). This is the start- 
ing point of our research. We can rewrite Eq. ([8]) as 



PHxl 



Ni 



P[X_ 



where 



a[XN] :--lnPj[4]+lnPo[a;o 



(9) 



(10) 



k=0 



which is called the entropy production along trajectory 
Xn- 

Various proofs of the detailed fluctuation theorem 
[Eqs. ([8]) and ([9|)] for stochastic systems have been 
presented, for example, in Refs. [l3, [ill for the 
Markovian stochastic dynamics and in Ref. [28| for non- 
Mar kovian Langevin systems. A proof of Eqs. ([8]) and 
^ has also been given in Ref. [1^] for the situations in 
which the total system including heat baths is treated 
as a Hamiltonian system and the initial states of the 
heat baths in the forward and backward processes are 
the canonical distributions. This proof can conflrm the 
physical validity of the detailed fiuctuation theorem even 
for the non-Markovian dynamics with multi-heat baths, 
as the stochastic dynamics can be reproduced as that of 
a partial system of the total Hamiltonian system includ- 
ing the heat baths. We also note that several equalities 
that are similar but not equivalent to Eqs. ([8]) and ^ 
have been derived for different situations. For example, 
the transient fluctuation theorem has been discussed for 
dynamical systems in Ref. [l^ . The fluctuation theorem 
for nonequilibrium steady states has been discussed for 
stochastic systems 9, 12] and dynamical systems [1, 0|- 

From the detailed fiuctuation theorem ^ , we can show 
Crooks' fiuctuation theorem as follows. We denote as 
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P[cr] the probability of finding the entropy production a 
in the forward processes, satisfying 



P[c7] = J S{a-a[XN])P[XN]dXN, 



(11) 



where 5{-) is the delta function. On the other hand, let 
P^[cr] be the probability of obtaining a in the backward 
processes, satisfying 

P^[a] = j a[X^])pt[x^]dx^. (12) 

By using the detailed fluctuation theorem ([9]) and equal- 



ity a[XN] 
orem 



-(t[x]|^], we obtain Crooks' fluctuation the- 



P\(J] 



(13) 



The detailed fluctuation theorem ([9]) or Crooks' fluctu- 
ation theorem (fT3|) leads to the integral fluctuation the- 
orem 



(e-^) = 1, 



(14) 



where the ensemble average (• • • ) is taken over all tra- 
jectories under forward protocol (see Eq. (jS])). From the 
concavity of the exponential function, we obtain 



{<y) > 0, 



(15) 



which is an expression of the second law of thermodynam- 
ics: the ensemble-averaged entropy production is non- 
negative. By taking the ensemble average of the loga- 
rithm of both sides of Eq. we have 



cLXnPIXn] In 



P\X 



(16) 



which we will refer to as the Kawai-Parrondo-Broeck 
(KPB) equality 29,,^]. The right-hand side of Eq. ^ 
is the KuUback-Leibler divergence (or the relative en- 
tropy) of P[Xjv] and P^[X]y], which is always positive. 
Therefore, Eq. ((TB)) reproduces inequality (|15p . 

If the probability distribution of a is Gaussian, the 
cumulant expansion of Eq. ^ leads to a variant of 
fluctuation-dissipation relation 



(17) 



which indicates that (cr) is determined by the fluctuation 
of <T. Equality (I17p is an expression of the fluctuation- 
dissipation theorem of the first kind, which gives a special 
case of the Green-Kubo formula |19l ]. 

In the case of an isothermal process with a single heat 
bath, the entropy production reduces to 



a[XM] = ^{W[Xn] - AP), 



(18) 



where M^[X7v] is the work performed on the system dur- 
ing the process, and AF is the difference of the free en- 
ergies for the initial and final Hamiltonians (see also Ap- 
pendix A for details) . Under this situation, Eq. (ITi)) leads 
to 



-PAF 



which is the Jarzynski equality [8 
thermodynamics then reduces to 

{W) > AF. 



(19) 

The second law of 
(20) 



III. MEASUREMENT 

In this section, we formulate and investigate the effect 
of measurements on nonequilibrium dynamics. 

A. Classical Measurement and Mutual Information 

In this subsection, we review the general framework 
of a measurement on a probabilistic variable, which can 
be applied to a broad class of measurements on classical 
systems. 

Let X be an arbitrary probability variable of a mea- 
sured system whose distribution is P[x]. We perform a 
measurement on it and obtain outcome y which is also 
a probability variable. The error of the measurement 
can be characterized by a conditional probability P[?/|a;], 
which describes the probability of obtaining outcome y 
under the condition that the true value of the measured 
system is x. We note that J2y ^ivl^] — 1 lo'" where 
we note that the sum should be replaced by the inte- 
gral if y is a continuous variable. If the measurement 
is error- free, P[y|a;] is given by the delta function or the 
Kronecker's delta. We assume that P[2/|a:] is independent 
of the probability distribution P[x]; in other words, the 
error is independent of the state preparation of the mea- 
sured system. The joint probability of x and y is given 
hy P[x,y] — P[y\x]P[x], and the probability of obtaining 
y by P[y] — X^k^!'^'?/]- The probability of realizing x 
under the condition that the measurement outcome is y, 
denoted as P[a;|y], is given by the Bayes theorem: 



P[x\y] - 



P[y\x]P[x] 

P[y] 



(21) 



We next discuss th e inf ormation contents related to 
the measurement [99l Il00l |. The Shannon information 
contents of the probability variables are given by 

^-=-E^MlnPM, i/, :=-^PMlnP[y], (22) 



which characterize the randomnesses of x and y, respec- 
tively. On the other hand, the mutual information con- 
tent (/) between x and y is given by 



(/) :=^P[x,y]/[x:y], 



(23) 



xy 
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where 



I[x : y] 



In 



P[y\a 
P\y] 



(24) 



In this paper, we also call I[x : y] the mutual information. 
We note that I[x : y] ~ I[y : x] holds due to the Bayes 
theorem ((2T|) . 

The mutual information (/) measures the amount of 
information obtained by the measurement. It is known 
that 

< (/) < H^, < (/) < Hy. (25) 
If the measurement is error-free, (/) = = Hy holds. 

B. Measurements on Nonequilibrium Dynamics 

We next formulate multiple measurements on nonequi- 
librium dynamics, and discuss the properties of the mu- 
tual information obtained by the measurements. 

Let y„ be the outcome at time i„ := nAt. In this 
section, we assume the foUowings: 

1. The error of the measurement at time i„ is char- 
acterizes by P[?/„|X„], where ?/„ can depend on 
the trajectory of the system before t„ due to the 
causality. Here we assumed that the property of 
the measurement error at time i„ does not explic- 
itly depend on or P[Xn\. This assumption is 
also justified in many real experimental situations. 

2. The unconditional probability distribution of Xn, 
P[Xn], is not affected by the back-action of the 
measurement. Since the system is classical, this 
assumption is justified for many real systems such 
as colloidal particles and macromolecules. 

If P[yn\Xn] = P[yn\xn], wc Call the measurement 
Markovian, which means that the outcome is determined 
only by the system's state immediately before the mea- 
surement. This condition is satisfied if the measurements 
can be performed in a time interval that is sufficiently 
shorter than the shortest time scale At of the system. 
We note that the Markovness of the measurement is in- 
dependent of that of the dynamics. 

We assume that the measurements are performed at 
times tni, tn2, ■ ■ ■ , 1 where < rii < n2 < • • • < 
< N . If m = 0, ri2 = 1, = 2, • • ■ , njv+i = N 
hold, the measurement is time-continuous in the limit of 
At — s- 0, because the measurements are performed at all 
times. 

We write as F„ the set of measurement outcomes that 
are obtained up to time t„, i.e. F„ := [ym , 2/n2, ' • ' , V[n\) 
where [n] is the maximum Uk satisfying < n. If the 
measurement is continuous, then Yn = {yo, ■ • ■ , Vn)- 

We define 



M' 



(26) 



where M' is the maximum integer satisfying um' ^ n. 
Without feedback, Eq. ([^5]) defines the conditional prob- 
ability of obtaining outcomes Yn under the condition of 
Xn , while, with feedback, this interpretation of Eq. ([25)1 
is not necessarily correct as shown in the next section. 
To explicitly demonstrate this point and to distinguish 
Pc[^n|^n] from the usual conditional probability, we put 
sufhx "c" . Then, the joint distribution of X„ and Yn is 
given by 

P[Xn,Yn] = P,[Yn\Xn]P[Xn]. (27) 

The probability of obtaining outcomes Yn is given by 



» M' 
P[Yn] = / dXnP[Xn,Yn] ^ ]]_ P[ynjYn,-l] 

k=l 



(28) 



P[yn\Xn,Yn-l] 



where the two equalities are just identities known in prob- 
ability theory. We also note that 

P[Yn\Xn] 
P[Y„-i\Xn] (29) 
= P[yn\Xn], 

which is, in fact, independent of Yn-i- 

We then discuss the mutual information obtained 
by multiple measurements on nonequilibrium dynamics. 
Suppose that we obtain measurement outcomes Yn-i at 
time tn-i- If we perform another measurement at time 
tn and obtain outcome y„, we obtain the mutual infor- 
mation between j/„ and Xn under the condition that we 
have obtained Yn~i'. 



I[yn ■■ Xn\Yn-i] := In 



= In 



P[yn\Xn.Yn-l] 
P[Vn\Yn-l] 

P[yu\Xn] 



(30) 



P[yu\Yn-lY 

where we used Eq. ((29| . We note that, if the measure- 
ment is Markovian, /[y„ : X„|l^i_i] reduces to : 
We denote as Ic the sum of these mutual infor- 
mation contents obtained by multi-measurements, that 
is, 

M' 

h[Xn:Yn] ■.^Y.^[yn, : Xn,] 

k=l (31) 
P[Yn\Xn] 



In- 



P[Yn] ' 



fc=l 



where we used Eq. (|28p . We note that the same quan- 
tity has been discussed in Ref. [11]. From Eq. (PT|) . we 
find that Ic\Yn '■ Xn] equals the mutual information be- 
tween trajectories Xn and Yn defined as I[Yn : Xn] '■= 
ln{P[Yn\Xn]/P[Yn]). In the presence of feedback control, 
however, this is not true (i.e., Ic 7^ I), as we will see later. 



IV. FEEDBACK CONTROL 

In this section, we formulate feedback control on 
nonequilibrium dynamics. 
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A. Formulation 

Feedback control implies that protocol An depends on 
measurement outcomes (see FIG. 2). On the other 
hand, without feedback control, control protocols are pre- 
determined and independent of the measurement out- 
comes, as is the case for the setup of the original fluc- 
tuation theorem and Jarzynski equality. 



Control 
parameter 



X 



Thermodynamic 
system 

X 



Measurement 
outcome 



y 



Controller -4- 



FIG. 2: Feedback control on nonequilibrium dynamics. The 
control parameter is denoted as A, the point of the phase space 
of the system as x, and the outcome of measurement on the 
system as y. Parameter A depends on y through the real-time 
feedback control. 

When the system is subject to feedback control, A„ 
can depend on measurement outcomes that are obtained 
until tn, while A„ cannot depend on any measurement 
outcome that is obtained after time tn due to causal- 
ity. We introduce notation A„(y„), which means that 
the value of A at time t„ is determined by ¥„ . We write 
A„(y„_i) := (Ao(yo),Ai(yi),--- ,A„_i{K_i)). 

If A„ only depends on yn as A„(y„), the feedback proto- 
col is called Markovian. We note that the Markovness of 
feedback is independent of that of the dynamics or mea- 
surements. The Markovian feedback control is realized 
when the delay time of feedback is sufficiently smaller 
than the smallest time scale At of the dynamics. 



B. Overdamped Langevin System 

As a simple illustrative example, we discuss an over- 
damped Langevin system, whose equation of motion is 
given by 



V 



dx{t) _ dV{x,\) 



dt 



dx 



+ f{\) + ^2^m, (32) 



where "q is the friction constant, V{x^X) is an external 
potential, /(A) is an external nonconservative force, and 
f(i) is the Gaussian white noise satisfying {^{t)£,{t')) = 
6{t — t'). The detailed fluctuation theorem ([H]) is still sat- 
isfied in the presence of a nonconservative force that vio- 
lates the detailed balance, because Eq. ^ can be derived 
from the local transition rate of the stochastic dynamics 
and is independent whether there is a global potential or 
not. We assume that the measurement is time-continuous 
and Markovian: 



At 



(33) 



where Ai?„ is a white Gaussian noise with (Ai?„Ai?„/) = 
RSnn'At {R > 0). The conditional probability of obtain- 
ing outcome yn is given by 



P[yn 



oc exp 



(34) 



The feedback protocol can be written as 
\i{ya,yi, ■ ■ ■ ,yn) in general. The work performed 
on the system is then given by 



Wn := V{Xn,Xn+l) - V{Xn, A„) 

-o(At), 



5A^^" 



(35) 



where 

AA„ := A„+i(yo,yi, • • • ,yn+i) - A„(?;o,yi, • • • ,2/«). 

(36) 

In particular, if the feedback is Markovian, A„ is given 
by A„ (?/„). Then, AA„ = A„+i(y„+i) - A„(?/„) can be 
written as 



9X . dX , 1 S^A , 2 



(37) 



where A?/„ :— yn+i ~ yn- The first term on the right- 
hand side of Eq. (|37|) arises from a change in A by the 
pre- fixed protocol, while the second and third terms are 
induced by the feedback control. 

We next consider the Kalman filter and the optimal 
control. As a special case of Eq. ([32l) . we consider a 
discretized linear Langevin equation in the Ito form: 



-KXnAt + \nAt + y/2r]kBTAWn, (38) 



where K is a positive constant and A„ is a control pa- 
rameter. If the initial distribution of xq is Gaussian, the 
distribution of x„ remains Gaussian with Eq. (j38p . In 
this case, the obtained mutual information by measure- 
ment p3|) at time t„ is given by 



{Ic[Xn ■■ yn]) 



1 



■In 1 



Sn 



At 



Sn 



At+oiAt), (39) 



where Sn {xD - {xn)"^ 
information 



R J 2R 
. Therefore, the total mutual 



N 



(/c)- Inn Y^At 



N^oo ^ 2R 
n=0 



(40) 



can converge, while the measurement is continuous. 

We consider the Kalman filter on Eq. (|38l) with mea- 
surement (1331) . The Kalman filter is a standard method 
to construct the optimal estimator of a;„, denoted 
in terms of the mean square error. From measurement 
outcomes F„, Xn is obtained as the soluti on t o the fol- 
lowing simultaneous differential equations 10l| : 



Xn+l 



- ^^- + ^- At+^{yn-Xn)At, (41) 
r] R 
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A-n+l — An — 



-2KAn + 2kBT _ An 
rj R 



At, (42) 



where An is a time-dependent real number and Eq. (j42l) is 
a discretized version of the Riccati equation. By usin g the 
Kalman estimator Xn , the optimal control protocol [102j | 
is given by 



A. 



(43) 



where C„ is a pre-determined constant depending on the 
target of the optimal control. We note that the optimal 
control is a non-Markovian control as A„ = A„(l^), be- 
cause we use all of Yn = (jjo, j/i, • • • , j/n) to calculate a:„. 
The generalized Jarzynski equality for this situation has 
been discussed in Ref. 16411. 



C. Probability Distributions with Feedback 

We discuss the probability distributions with feedback 
control in general. Under the condition that we fix con- 
trol protocol An{Yn) with Yn being fixed, the condi- 
tional probability of realizing Xn is given by 

fe-i 

P[Xn\An{Yn-l)] = Po[^o] H P[xk+l\Xk, XkiYk)], 

k=0 

(44) 

which corresponds to Eq. ([1]). We note that, in the 
expression in Eq. (j44p . we do not omit the notation 
AAr(y/v_i) because its YAr_i-dependence is crucial. We 
also write 

fc-i 

P[Xn\xo,AniYn-i)] := [| P[xfc+i |Xfe, Afe (Ffe)] . (45) 

fc=0 

On the other hand, along the trajectory X„, the condi- 
tional probability of obtaining outcome j/„ at time tn is 
written as P[?/„|X„]. We then define 



Pc[Yn\Xn] -.^ II P[yk\Xkl 



(46) 



k=0 



which is to be compared with Eq. ([26)1 . 

We then obtain the joint probability distribution of Xn 
and Yn with feedback control as 



n-l 



P[Xn,Yn] = Y[ P[yk+l\Xk+l]P[xk+l\Xk,\kiYk- 



k=0 



= P4Yn\Xn]P[Xn\An{Yn-l)]. 

We can check that 



J P[Xn,Yn]dXndYn = 1, 



(47) 



(48) 



by integrating X„ and Yn in Eq. (|T7)) in the order of 

yn Xn ^ y-n-l Xn-1 ~>- ■ ■ ■ ^ yi Xi yo Xq, 



where the causality of measurements and feedback play 
crucial roles. 

The marginal distributions are given by 

P[Xn]= J P[Xn,Yn]dYn, P[Yn] = J P[Xn,Yn]dXn, 

(49) 

and the conditional distributions by 

P[Xn\Yn] = P[Yn\Xn] = ^^^^ ■ (50) 

We stress that, in the presence of feedback control, 

P[Yn\Xn] ^ Pc[Yn\Xn] (51) 

in general, because protocol An depends on Yn-i- On 
the other hand, without feedback control, P[y„|X„] = 
Pc[in|-''^n] holds because P[X„] is simply given by 
P[X„|A„] with A„ being independent of F„. 

The ensemble average of a probability variable 
A[Xn,Yn] is given by 

(A):- J A[Xn,Yn]P[Xn,Yn]dXndYn, (52) 

and the conditional average under the condition of Yn is 
given by 



{A)y„:= J A[Xn,Yn]P[Xn\Yn]dXn. 



(53) 



Equation (|^ still holds in the presence of feedback con- 
trol: 



P[yn\Xn,Yn-l] : = 



P[Xn, Yn] 
P[Xn, Yn-l] 

P4Yn]Xn]P[Xn\An{Yn-l)] (54) 
Pc[Yn-l\Xn-l]P[Xn\An{Yn-l)] 
= P[yn\Xn]. 

We note that Eq. (1^5)) also holds with feedback control. 

We then define the mutual information in the same 
way as in the case without feedback control: 

M' 

h[Yn:Xn] :-^/[y„, :X„jy„,_i] 

k=i (55) 

Pc[Yn\Xn] 



= In- 



P[Yn 



In the presence of feedback control, Ic[Yn : X„] does not 
equal the mutual information between trajectories Xn 
and Yn defined as /[r„ : X„] := ln(P[r„|X„]/P[r„]), 
because Pc[y,i|X„] ^ P[y,i|X„]. Intuitively speaking, Ic 
only characterizes the correlation between Xn and Yn due 
to the measurements, while / involves the correlation due 
to the feedback control. Note that 1^ is a more impor- 
tant quantity than /, because Ic has a clear information- 
theoretic significance: Ic is the information that we ob- 
tain by measurements. We also note that, in the case of 
a single measurement and feedback, Ic = I always holds. 
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We also note that an identity similar to the integral 
fluctuation theorem holds for Ir'. 



because 



{e-'') - 1, 

fc[yN\^N\ 

dXNdYNP[YN]P[XN\KN{YN-i)] = 1. 



(56) 



(57) 



D. Detailed Fluctuation Theorem for a Fixed 
Control Protocol 

If we fix control protocol AAr(l7v) with Y/v being fixed, 
then the detailed fluctuation theorem (ISl) still holds: 



P[X],\xI,Kn{Yn-i) 

P[XN\xo,kN{Yl 



N-l) 



exp [J2|3^Q^[XN,ANiYN^l)] 

(58)' 



where 



An{Yn)^ ■■= {XN-liYN-lY 



,MYor)- (59) 



The left-hand side of Eq. ([55)) corresponds to the fol- 
lowing forward and backward experiments. We first 
perform forward experiments many times with feedback 
control, and choose the subensemble in which the mea- 
surement outcomes are given by Yn-i- Within this 
subensemble, the ratio of trajectory Xn is given by 
P[Xn\xo, -^-NiYN-i)] under the condition of initial xq- 
We next perform backward experiments with protocol 
^NiYN-iY , where Y^^i was chosen in the forward 
experiments. We stress that we do not perform any 
feedback in the backward experiments: Ajv(iAf-i)^ is 
just the time- reversal of Ajv(YAr-i)- We then obtain 
P[xIj\xq, Af:[{Y]Y_i)^ as the ratio of trajectory Xj^, un- 
der the condition of initial Xq in the backward exper- 
iments. The original detailed fluctuation theorem ^ 
can straightforwardly be applied to this subensemble cor- 
responding to y/v-i because we have a unique control 
protocol in the subensemble, and therefore we obtain 

Eq. dsg). 

Let the initial distribution of the backward experi- 
ments be PQ[a;J|YAr] which in general depends on the 
measurement outcomes in the forward experiments. 
A natural choice of Pg^[a;J|lAr] is a stationary state 
Ps[xl\X{YNy]. Then we have 



P[Xj,\ANiYN)^ 



exp(-a[Xjv,Ajv(rjv)]), (60) 



P[XN\ANiYN)] 

where 

a[XN , An (Yn)] := - In Pj [4 1^"^] + In Po [xo] 
-^AQ,[AAr,Ajv(lV-i)]. 



If there is a single heat bath and the initial distributions 
of the forward and backward experiments are given by 
the canonical distributions, then the entropy production 
reduces to 

a[XN,AN{YN)] = f3{W[XN,AN{YN)] - AF[Yn]), (62) 

where the free-energy difference can depend on the mea- 
surement outcomes as AF[lAr] := F{Xfin{YN)) — P(Aint). 
We list our notations in Table I. 



X 


Phase-space point of the system. 


X* 


Time-reversal of x. 


t„ := nAt 


Discretization of time. 




Phase-space point at time t„. 




Trajectory of the phase-space point from 
time to i.e. X„ ~ {xo, Xi, - ■ ■ , Xn). 


xi 


Time-reversal of Xn, i.e. : = 

(Xq, X-^, ■ ' " , ^ji) with X^ . — Xj\i_n- 


Xn 


Controllable external parameters at time 

in . 


An 


Control protocol from time to 


Al 


Backward control protocol of A„, i.e. 

Ai ■- (aj,a|,--- ,xi) with xi ■- 

'^N-n-l- 


a[Xn,An] 


Entropy production. 


Vn 


Measurement outcome at time tn- 


Yn 


Measurement outcomes from time to 


n 


Time-reversal of Yn, i.e. {yl,yl,--- ,yi) 
with yi ■- ylr^n- 


An{Yn-l) 


Protocol of feedback control with outcomes 

Yn-l. 


P\XJAn(Yn]] 


Probability of trajectory Xn under the 
condition that the protocol is given by 
An{Yn) with Yn being fixed outcomes. 


A„(y„_i)t 


Time-reversal of A„(Y'„_i): A„(Y'„_i)^ : = 

{Xn-i{Yn^i)* , • ■ • , AjV_„_l (yAT-n-l)*). 


P[yn\Xn] 


Probability density of obtaining y„ under 
the condition of X„, which characterizes 
the measurement error. 


Pc [Y„ \X„] 


:= U,P[ynJXn,]. 


P[Xn,Yn] 


Joint distribution of Xn and Yn 
which is given by P[X„,'K„] = 

Pc[Yn\Xn]P[Xn\An{Yn-l)]. 


I[yn ■■ Xn\Yn-l] 


Conditional mutual information obtained 
at time t„ under the condition that we have 
outcomes Yn-i- 


Ic[Xn '■ Yn] 


Sum of the conditional mutual infor- 
mation: Ic[Xn ■■ Yn] ■— rifc-^bifc ■ 
XnJYn,-,]. 



V. 



TABLE L Symbols and their meanings. 



NONEQUILIBRIUM EQUALITIES WITH 
FEEDBACK CONTROL 



We now discuss the main results of this paper. We 
derive the two types of the generalized nonequilibrium 
equalities with feedback control in Sec. V A and V B, 
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respectively. The former generalization involves the mu- 
tual information, while the latter involves the efficacy of 
feedback control. 



A. Generalized Fluctuation Theorem with Mutual 
Information 

To derive a generalized detailed fluctuation theorem, 
we first formulate the relevant backward probabilities. 
We consider the following type of "backward probability 
distribution" : 



which satisfies 



J PHxl,,YN]dXldY^ = l. 



(63) 



(64) 



Definition (j63|) has a clear operational meaning. Suppose 
that we perform a forward experiment with feedback and 
obtain outcome Y^. We then perform a backward exper- 
iment with protocol Ajv(yAr-i)^- We repeat this set of 
the forward and backward experiments many times, and 
calculate the fractions of {Xn,Yj^) and (xj^, F^v), which 
respectively give PIX^^^Y^] and P^Xl^,Y]y]. 

Noting Eq. (|47p and the definition of the mutual infor- 
mation (|55p , we obtain a generalized detailed fluctuation 
theorem with feedback control: 



P^xl^Yj 



P[Xn,Yn] 



= cxp{-a[XN,AN{YN)] - Ic[Xn : Yj 



(65) 

where the effect of feedback control is involved by the 
term of the mutual information that is obtained in the 
forward experiments. We stress that, to obtain Eq. (1551) . 
we do not perform feedback control in the backward ex- 
periments. We just reverse forward protocol as Eq. (|59|) 
in the backward experiments. The same result for a spe- 
cial case was obtained in Ref. [g^]. The investigation 
of the detailed fluctuation theorem in the situations in 
which feedback control is also performed in the backward 
processes 66] is an interesting future challenge. Such sit- 
uations would be relevant to, for example, autonomous 
systems consisting of the controlled system and the con- 
troller, in which feedback control should also be needed 
for the backward processes. We can expect that the back- 
ward processes with feedback control can be used to char- 
acterize the reversibility of the autonomous systems. 

From the generalized detailed fluctuation theorem 
(I55|) . we obtain a generalized integral fluctuation theo- 
rem 16311: 



(66) 



Due to the concavity of the exponential function, we 
obtain a generalized second law of thermodynamics (63l . 
SI: 



which means that the entropy production can be negative 
due to the effect of feedback control (or due to the action 
of Maxwell's demon), and that the lower bound of the en- 
tropy production is bounded by the mutual information 

The reason why the entropy production can be neg- 
ative is that one can rectify the thermal fluctuations 
by feedback control. This negative entropy production 
is compensated for by the excess entropy production in 
the demon or the feedback controller [9a], and therefore 
the entropy production in the total system consisting of 
the demon and the information heat engine is consis- 
tent with the second law of thermodynamics. The key 
feature of feedback control is that it enables us to con- 
trol the entropy production of a partial system by uti- 
lizing the mutual information beyond the limitation of 
the conventional thermodynamics. Inequality (j67p iden- 
tifles the lower bound of the entropy production with 
feedback control, which plays a role parallel to the con- 
ventional second law of thermodynamics that gives the 
lower bound of zero in the absence of feedback control. 
Therefore, inequality (1671) is regarded as a generalization 
of the second law of thermodynamics that can be applied 
to feedback-controlled processes. 

We also obtain, by taking the ensemble average of the 
logarithm of the both sides of Eq. that 



(a) + (/c) = / dXNdYNP[XN,YN]\n 



P[Xn,Yn] 

pHxI^YnY 

(68) 

which is a generalization of the Kawai-Parrondo-Broeck 
(KPB) equality (HH). We note that the right-hand 
side of Eq. is positive because it is the Kullback- 
Leibler divergence between two probability distributions 
P[XN,Y]\r] and P^X^^jY^]; thus, inequality ([ST]) is re- 
produced. We note that equality in ([S7)) is achieved if 
and only if ct -|- /c does not fluctuate, or equivalently, if 



P[Xn,Yn]=PHxI„Yn] 



(69) 



holds, which implies the reversibility with feedback con- 
trol (70i] . The more the probability distribution of the 
forward processes with feedback is different from that of 
the backward processes without feedback, the more (a) 
is different from — (/c)- 

If the joint distribution of a and Ic is Gaussian, we 
have a generalized fluctuation-dissipation theorem from 
the second cumulant of Eq. : 



1 



(70) 



(^) > -{Ic), 



(67) 



which suggests that there is a trade-off relation between 
the entropy production and the mutual information. 

For the case in which a = I3{W — AF), Eq. (p5)) leads 
to a generalized Jarzynski equality: 

^g-,3(W-AF)-/.^ = 1, (71) 



(72) 



and inequality (|67|) leads to 

{W) > AF - A:bT(/c 
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We note that Eq. ([71]) and inequality ([72l) are the general- 
izations of the results obtained in Refs. t63|,l62i]. By defin- 
ing Wcxt '■= —W and setting AF = 0, we may rewrite 
inequahty ([7^ as 



(73) 



which implies that we can extract a positive work up to 
the term that is equal to the mutual information multi- 
plied by A:bT, from a thermodynamic cycle with a sin- 
gle heat bath with the assistance of feedback control 
or Maxwell's demon. The mutual information can be 
used as a "resource" of the work or the free energy. 
In the case of the Szilard engine, (/c) = (/) = In 2 
and (Wcxt) = k-QT h\2 hold, and therefore the equal- 
ity in (I73|) is achieved. In fact, in the Szilard engine, 
(J + I — (3{W — AF) + I does not fluctuate, but is zero 
for both outcomes "left" and "right." 

We note that, to obtain Eq. or ([7T|) experimentally 
or numerically, the condition of Pc[i^Af l-'i^Ar] ^ needs to 
be satisfied for all {Xn,Yn). To explicitly see this, we 
write Pc[i^Af l-'^Af] =: e > 0. We then obtain 



P[XN,YN]e- 



-a-Ic 



eP[XN] ■ -e 

e 



-a+\nP[YN] 



(74) 



which does not converge to zero with the limit of £ — 0. 
On the other hand, in real experiments or numerical sim- 
ulations, the events with P[Xjv,Y/v] — never occur. 
Therefore, if Pc[Yn\Xn] = holds for some {Xn,Yn), 
the terms associated with zero-probability events make 
non-zero contributions to Eq. (1661) and (j71l) : in such cases, 
we cannot obtain Eq. (|66| or (|7ip experimentally or nu- 
merically. On the contrary. 



P[Xn,Yn]I: =eP[XN] ■ In 



P[Yn] 



(75) 



converges to zero for all n = 1,2,---, in the limit of 
e ^ 0. Therefore, we can find (/") experimentally and 
numerically even if Pc[YAf|Xjv] = for some {X^^Yn), 
and also obtain Eqs. (|68|) . ((TO)) , and inequalities (l67t . 



B. Generalized Fluctuation Theorem with Efficacy 
Parameter 



We next derive a different type of nonequilib- 
rium equality. In this subsection, we assume that 
the measurements are Markovian (i.e., P[j/n|X„] = 
P[?/ri|a;„] holds). We perform forward experiments 
with measurements at times t„j , t„2, • • • , tn^^ with feed- 
back control, and perform backward experiments with- 
out feedback but only with measurements at times 



Let Y' 



[y'N-nM^y'N-nM-V--- ^y'N-nJ be the 

measurement outcomes in the backward measurements. 



N 



Then, the probability of obtaining under the condi- 
tion of X\j is given by 



M 



Pe[r^|X^] ■■^\{P[y'N-n^^^N-u^■ 



(76) 



fc=l 



Therefore, the probability of obtaining Y^ under protocol 

A(rw)t is 

P[Y^,\Kn{Yn-i)^ - j P,[Y^,\X^^]P[X^^\KN{YN-i)^dX 

(77) 

which is normalized as 



j P[Yi,\kN{YN-i)^dY'^ 



1, 



(78) 



where the probability variable is independent of Yat. 

We then consider the time-reversed sequence of Yat. 
Let y* be the time-reversal of for example, if we 
measure the momentum, then y* —yn- We write 



Y 



N 



yN -nil 



^y*N~m)- The probability 



of Y^ = Y^ in the backward experiments is given by 

P[Y^|AAr(YAr„l)t] = j Pe[Y^|Xt,]P[X^|Aw(Y^_i)t]dX 

(79) 

which is the probability of obtaining the time-reversed 
outcomes by time-reversed measurements during the 
time-reversed protocol. We stress that 



j P[Y^|AAr(Yjv_i)t]dY^ ^ 1 



(80) 



in general because Y^ is no longer independent of Y^v-i. 

In the following, we assume that the measurements 
have the time-reversed symmetry 



P[y:i<J = P[y„|x„]. 
for all n, which leads to 

Pc[Y„t|Xt]=P,[Y„|X„ 



(81) 



(82) 



We then have the "renormalized" (or "coarse-grained") 
detailed fluctuation theorem [2^, H^l 



P[Y^|A(YA.)t] 



P[Yi 



(83) 



where cr' [Yat] is the "renormalized" (or "coarse-grained") 
entropy production defined as 



a' [Yat] := - ln(e-'^)y„ 



= -ln j dXjve-'^[-^"^'^"(^"-i)lp[XAr|YAr]. 



(84) 



Equality ([83|) implies that the detailed fluctuation theo- 
rem retains its form under the coarse-graining, if we in- 
troduce the appropriate coarse-grained entropy produc- 
tion. From the concavity of the exponential function, we 
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obtain cr'[Y/v] < {<^)yn and {a') < (a). The same result 
for a different setup has been obtained in [2^ [s^ . 

The proof of ([83| goes as follows. From the definition 
of cr'[y/v] and the detailed fluctuation theorem ([55]). we 
have 



dX 



N 



-P[Xn\Yi 



= / dX 



N- 



P[xj,\AN{Yr,^ 
P[Xn\An{Yn-i)] 

P[Xj,\ANiYN-iy]P[XN,YN] 



P[Xn\An{Yn-i)] P[Yn] 

/ dXNP[Xl\AN{YN-iy]Pc[YN\X 

-pi^ I dXNP[xUAN{YN^iymYi\xi 



(85) 



In the last line, we used the time-reversal symmetry 
of the measurements. By noting Eq. (l79t . we obtain (l83l) . 

We note that Eq. (|83p holds regardless of the presence 
of feedback control. Without feedback control, Eq. 
reduces to 



P[Yi 



N 



(86) 



By taking the ensemble average of both sides of 
Eq. (IH31) and noting that (e"'^') = (e"'^) holds, we ob- 
tain the second generalization of the integral fluctuation 
theorem [6J 



7, 



(87) 



where 7 is the efficacy parameter of feedback control de- 
fined as 



7:= / P[4|AAr(lW_i)t]d4, 



(88) 



which is the sum of probabilities of obtaining the time- 
reversed outcomes by the time-reversed measurements 
during the time-reversed protocols (see FIG. 3). If 
cr = I3{W - AF) holds, Eq. (HH) leads to the second 
generalization of the Jarzynski equality (63| : 



(89) 



If the feedback control in the forward processes is "per- 
fect," the particle is expected to return to its initial 
state with unit probability in the backward processes. 
In such a case, 7 takes the maximum value that equals 
the number of possible outcomes of Y/v. In fact, for the 
case of the Szilard engine, 7 = 2 holds corresponding to 
W = -knT \n2 and AP = [11]. In contrast, without 
feedback control, 7 reduces to 1 as 



7 1 P[Y^]dY^ - 1, 



(90) 



which vindicates the original integral fluctuation theo- 
rem. Therefore, the measurements in the backward pro- 
cesses are used to characterized to the efficacy of feedback 
control in the forward processes. 



t. ?„ 



■t t 



A.,(Fv)' 



(a) 



(b) 



FIG. 3: (a) Forward outcomes Yn with forward protocol 
Ajv(yiv). (b) Backward outcomes with backward protocol 
{An{YnV). 



We stress that a and 7 can be measured indepen- 
dently, because a is obtained from the forward exper- 
iments with feedback and 7 is obtained from the back- 
ward experiments without feedback. Therefore, Eqs. (j87p 
and (j89p can be directly verified in experiments. In fact, 
Eq. (|89p has been verified in a real experiment by using a 
feedback-controlled ratchet with a Brownian particle [731 ■ 

From Eq. (p5|) . we have the second generalization of 
the second law of thermodynamics 



(cr) > — In 7. 



(91) 



The equality in inequality (|9ip is achieved if a does not 
fluctuate. We note that, if the distribution of a is Gaus- 
sian, we have a generalized fluctuation-dissipation theo- 
rem 



{a)+\n^ = \{{cy^)-{af). 



(92) 



While the first generalization (j66| only involves the 
term of the obtained information, the second general- 
ization (|87p involves the term of feedback efficacy. To 
understand the relationship between the mutual infor- 
mation /c and the feedback efficacy 7, we introduce the 
notation 



C[A] 



ln(e-^) 



(93) 



for any probability variable A. We note that, if A can 
be written as A — tA' with t being a real number and A' 
being another probability variable, then C[A] is the cu- 
mulant generation function of A' . By using this notation, 
we have 



C[a]+C[I,]-C[a + Ic]=-\n-f, 



(94) 



because C[cr] = — In 7 in Eq. ([57)1 . C[Ic] = holds as in 
Eq. (HH), and C[(t + /c] = holds as in Eq. ([Ml)- Equality 
(j94| implies that — In 7 is a measure of the correlation 
between a and Ic- This can be more clearly seen by the 
cumulant expansion of Eq. (1941) if the joint distribution 
of a and h is Gaussian: 



In 7. 



(95) 



Therefore, 7 characterizes how efficiently we use the ob- 
tained information to decrease the entropy production by 
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feedback control: if 7 is large, the more Ic we obtain, the 
less a is. 

We can also derive another nonequilibrium equality 
which also gives us the information about the feedback ef- 
ficacy. By taking logarithm of the both sides of Eq. (|65l) , 
we obtain 

{a') ^ f dYNP[YN]\n (96) 

which is a generalization of Eq. (|16|) . The same re- 
sult under a different situation has also been obtained 
in Ref [H, Equality ^ implies that the renor- 

malized entropy production equals the Kullback-Leibler 
divergence-like quantity between the forward probability 

P[Yn] and the backward probability P[Y^|AAr(F7v-i)''']. 
In fact, without feedback control, the right-hand side of 
Eq. ([M)) reduces to the Kullback-Leibler divergence be- 
tween /"[i^Af] and P^[Y^] and therefore the both sides of 
Eq. (j96l) are positive, which is consistent with the second 
law of thermodynamics. On the contrary, in the presence 
of feedback control, the right-hand side is no longer the 
Kullback-Leibler divergence, because P[Y^|A7v(5^Ar-i)^] 
is not a normalized probability distribution in terms of 
Yjjf. Therefore the both sides of 1^ can be negative. 
Since (cr') < {a), the entropy production (a) is bounded 
from below by the right-hand side of Eq. (|96)) : 

(a) > [ dY^P[Y^] In ™ (97) 

J P[Yl\ANiYN-iy] 

Without feedback control, the right-hand side of (|97|) 
gives a positive bound, while, with feedback control, the 
right-hand side can give a negative bound. We note that, 
for a quantum generalization of the Szilard engine with 
multi-particles, essentially the same result as Eq. ([M)) has 
been obtained (98i|. 

We note that special cases of our results in this section 
were obtained elsewhere. We have derived two types of 
the generalized Jarzynski equality for the cases with a 
single measurement in the presence of a single heat bath 
in Ref. [gS^I . In Ref. [67| , the detailed fluctuation theorem 
and the Jarzynski equality were obtained for the cases 
with multi-measurements and feedback in the presence of 
a single heat bath. In Ref. \6^, a generalized Jarzynski 
equality was also obtained for the Kalman filter and the 
optimal control. The results in this paper include all of 
the above results, and generalize them to the cases of 
multi-heat baths and non-Markovian measurements. 

We also note that the generalized Jarzynski equal- 
ity (|89p with a single measurement was experimentally 
verified by using a feedback-controlled ratchet with a col- 
loidal particle [77]. Moreover, Eq. (15^1) has been general- 
ized to quantum systems [68| . 



VI. EXAMPLES 

We now discuss two examples which illustrate the es- 
sential features of our general results. We analytically 
discuss a generalized Szilard engine with measurement 
errors in Sec. VI A, and numerically discuss a feedback- 
controlled ratchet in Sec. VI B. 



A. Szilard Engine with Measurement Errors 

As an example with a classical measurement, we dis- 
cuss a generalized Szilard engine with measurement er- 
rors, which will be shown to achieve the upper bound of 
inequality ([7^ or ([75]) for an arbitrary error rate. The 
control protocol of the generalized Szilard engine is given 
by the following steps, which are described FIG. 4. 

Step 1: Initial state. A single-particle classical gas 
is in a box. The initial state of the gas is in thermal 
equilibrium with a single heat bath at temperature T = 
(fcB/3)-^ 

Step 2: Insertion of the harrier. We insert a barrier in 
the middle of the box, and divide it to two boxes with 
the same volume. Here, we do not know in which box the 
particle is. For simplicity of notations, we write "left" as 
"0" and "right" as "1." In other words, the position x of 
the particle is given by a; = or x = 1. We do not need 
any work during this process as proved in [98j . 

Step 3: Measurement. We measure the position of the 
particle. We assume that the measurement is equi valen t 
to the binary symmetric channel with error rate s [lOC| : 
the measurement outcome takes y = or 1, and the 
measurement error is characterized by conditional prob- 
abilities P[0|0] = P[l|l] - 1 = e and P[0|1] = P[1|0] = e 
with < e < 1. We note that x = y holds for the original 
Szilard engine without error {e — 0). 

Step 4- Feedback. We next move the position of the 
barrier quasi-statically and isothermally. The protocol of 
moving the barrier depends on measurement outcome y. 
Let vq {0 < vq < 1) and vi {0 < vq < 1) be real numbers. 
We assume that, after we move the barrier, the ratio of 
the volumes of the boxes is assumed to be : 1 — vq for 
?/ = 0, or 1 — TJi : vi for y — 1. We note that, in the 
case of the original Szilard engine, vq — vi — 1 holds. In 
this process, we extract the work from the engine. The 
amounts of the work are given by kBThi2vQ if (x.y) — 
(0,0), fcBTln2(l- wo) if ix,y) = (0,1), fcBPln2(l - vi) 
if {x,y) = (1,0), and kBT\n2vi if {x,y) = (1,1). The 
feedback protocol is characterized by vq and vi . 

Step 5: Removal of the harrier. We remove the barrier 
without any work. The engine then returns to the initial 
state. 

From the total process, we extract the average work 

(Wext) =A:BP(ln2+i^lnwo + ^ln(l-«o) 
+ t\n(l-vi) + —^\iivi). 
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step 1: 

Initial state 



Step 2: 

Insertion of the barrier 



Step 3: 
Measurement 

{x,y)= (0,0) 



(1,0) 



(0,1) 



(1,1) 



Step 4: 
Feedbacl^ 



V„ 1-V„ V„ 1-V(, 1-V| V| 1-V| V| 



Step 5: 

Removal of the barrier 



FIG. 4: Generalized Szilard engine with measurement error 
rate e, where x denotes the states of the system, and y denote 
the measurement outcomes. The control protocol is deter- 
mined by y. 



We note that AF^ = holds. We then maximize (Wcxt) 
under a given measurement error e by changing vq and 
vi. The maximum value of (Wcxt) is achieved when 

Wo = vi = 1 - £, (99) 

for which the maximum work is given by 

(Wcxt) = fcBT[ln2 + £ln£+ (1 -£)ln(l -e)]. (100) 

On the other hand, the mutual information of the binary 
symmetric channel is given by 

(/) =ln2 + eln£+(l-£)ln(l-e). (101) 

Therefore, we obtain 

(VFext) = k^T{I), (102) 

which means that the generalized Szilard engine achieves 
the upper bound of the extractable work ([7^ or ([75]) for 
any amount of the mutual information. 

We also check the generalized Jarzynski equalities in 
this model for arbitrary vq, ui, and e. We first note 
that I[x : y] is given by ln2(l — £) when (x, y) — (0,0), 
ln2£ when {x,y) = (0,1), ln2£ when {x,y) = (1,0), and 
ln2(l — £) when {x,y) = (1, 1). Therefore we obtain 



-I^w-I^^ _ t^o + (1 - vq) + (1 - vi) + vi 



1, (103) 



which confirms Eq. (j7ip . 

We next consider the second generalization (j89p of the 
Jarzynski equality. Corresponding to two measurement 



outcomes y = 0, 1, we have two backward control proto- 
cols as follows (see also FIG. 5). 

Step 1. Initial state. The initial state of the backward 
control is in the thermal equilibrium. 

Step 2. Insertion of the harrier. Corresponding to Step 
5 of the forward process, we insert the barrier and decide 
the box into two boxes, because the time-reversal of the 
barrier removal is the barrier insertion. Corresponding 
to ?/ = or y = 1 in the forward process, we divide the 
box with the ratio : 1 — f o or 1 — : wi , respectively. 

Step 3. Moving the barrier. We next move the barrier 
to the middle of the box quasi-statically and isothermally. 
This is the time-reversal of the feedback control in Step 
4 of the forward process. 

Step 4- Measurement. We perform the measurement 
to find in which box the particle is in. Corresponding to 
the backward protocol with y — 0, we obtain the out- 
comes of backward measurement y' = with probability 
P[y' = 0|A(y = 0)t] = wo(l-£) + (l-wo)£ and y' ^ 1 with 
probability P[y' = l|A(y = 0)'^] = vqE + (1 - vo){l - e). 
On the other hand, corresponding to the backward pro- 
tocol with y = 1, we obtain the outcomes of backward 
measurement y' = with probability P[y' = 0|A(?/ — 
lY] = Vie + (1 — vi){l — e) and i/ = 1 with probability 
P[y' - l\A{y = l)t] ^v,{l-e) + {l- vo)e. 

Step 5. Removal of the barrier. We remove the bar- 
rier and the system returns to the initial state. This is 
the time-reversal of the barrier insertion in Step 2 of the 
forward process. 



step 5: 

Removal of the barrier 



(y,y')= (0,0) 



(0,1) 



(1,0) 



(1,1) 



Step 4: 
Measurement 



Step 3: 

Movement of the barrier 



r = 



l-V.l-l', V, 



Step 2: 

Insertion of the barrier 

Step 1: 
Initial state 



FIG. 5: Backward processes of the generalized Szilard engine. 
Corresponding to y that denotes the measurement outcomes 
in the forward process, we have two control protocols in the 
backward process, where y' denotes the measurement out- 
comes in the backward process. 
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(104) 



From Step 4 of the backward process, we have 

7 PW - 0|A(j/ = 0)t] + P[y' = l\K{y l)t] 
= (1 - £){vo + vi) + e(2 -vo- vi). 

On the other hand, we can straightforwardly obtain 

(e--^^) = (1 - e)(«o + ^^i) + £(2 - f o - (105) 

which confirms Eq. 



B. Feedback-controlled Ratchet 

_We next discuss a model for Brownian motor s |l03l- 
108 1. in particular a feedback-controlled ratchet [Sa. ISSl. 
60l |. We consider a rotating Brownian particle with a 
periodic boundary condition. Let x be the position 
or the angle of the particle, and its boundary condi- 
tion is given by x = x -\- L with L being a constant. 
In the following, we restrict the particle's position to 
—L/2 < X < L/2. We assume that the particle obeys 
the overdamped Langevin equation Eq. ([5^ . and that 
control parameter A takes two values (A = or 1). Cor- 
responding to them, the ratchet potential V takes the 
following two profiles (FIG. 6): 



V{x,0) = 



V{x,l) = 



K{x + L/2)/l 

{-L/2 <x < -L/2 + 1), 
-K{x-L/2)/{L-l) 

{-L/2 + 1 <x< L), 

-K{x + L/2-l)/{L-l) 

{-L/2 <x< -L/2 + 1), 

K{x + L/2 + 2l)/l 

{-L/2 + 1 <x< -L/2 + 21) 

-K{x- L/2-l)/{L-l) 

{-L/2 + 21 <x < L/2), 



(106) 



(107) 



where I is a constant with < I < L/2, and K is a, 
positive constant that characterizes the height of the po- 
tential. 





K 


\x = o 






-L/2 -L/2 + 1 





L/2 







K 


\x = i 






-L/2 f \ 





L/2 



-L/2 + 1 - L/2 + 21 



FIG. 6: Two shapes of potential V{x, A) corresponding to 
A = 0, 1. 



2. Flashing ratchet. At times t = rriTQ with m being 
integers and tq being a constant, we switch param- 
eter A from to 1 or from 1 to periodically. 

3. Feedback- controlled ratchet. At times t = rriTQ, we 
switch the parameter with the following feedback 
protocol. We measure the position x at t — totq 
without error. We then set A = 1 from t — mro to 
(m + 1)tq if and only if the outcome is in —L/2 < 
X < —L/2 + I. Otherwise, parameter A is set to 0. 

For numerical simulations, we set I = 3L/10, K — 
SfceT, To = 0.05, and r = 0.25, with units k^T = 1, 
L = 1, and 77/2 — 1. We performed the simulations by 
discretizing Eq. ([32]) with At = 0.00025 for 1,000,000 
samples. We note that, to obtain the initial thermal 
equilibrium, we waited r^ait =0.5 and checked that the 
system was fully thermalized in the periodic ratchet with 
parameter A = 0. 

The time evolution of the ensemble average {x{t)) is 
plotted in FIG. 7 (a) for the above three protocols. As 
expected, nothing happens for the first protocol, while 
the particle is transported to the right on average for the 
second and third protocols. In the case of the feedback- 
controlled ratchet, the particle is transported to the right 
faster than the case of the flashing ratchet. Figure 7 (b) 
shows the time evolution of the work {W{t)) that is per- 
formed on the particle. The work is induced only in the 
switching times. We find that, in order to transport the 
particle, the energy input to the particle with feedback 
control is smaller than that with the flashing. 



Feedback 



(a) 




0.10 0.15 0.20 

time t 



(b) 



Flashing 



Feedback 



0.10 0.15 

time t 



We start with the initial equilibrium with parameter 
A = 0, and control the system from time i = to r with 
the following three protocols. 

1. Trivial control. We do not change the parameter 
A = 0. 



FIG. 7: (a) Numerical results of the ensemble average of 
trajectory x{t) corresponding to the three control protocols: 
the trivial control, the flashing ratchet, and the feedback- 
controlled ratchet, (b) Numerical result of the ensemble av- 
erage of the work W{t) corresponding to the flashing ratchet 
and the feedback-controlled ratchet. 
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Figure 8 shows the left-hand side of the Jarzynski 
equality (e~''(^~^^)) for the flashing and feedback- 
controlled ratchet, and the efficacy parameter 7 for the 
feedback-controlled ratchet. We note that AF = al- 
ways holds. With feedback control, (e"^'-^"^^-') in- 
creases from 1 as the number of switchings increases, 
while, without feedback control, (e^'''^^^'^^^) converges 
to 1 for all switching times in consistent with the original 
Jarzynski equality. On the other hand, to obtain 7, we 
numerically performed the backward experiments. The 
discretization of the time is At = 0.0005, and the number 
of the samples is 10,000 for each trajectory of X{t). We 
note that the number of the trajectories of A is given by 
2™ with m times of switchings. Figure 8 shows a good 
coincidence between {e~^^) and 7, which confirms the 
validity of Eq. ([M)) in the feedback-controlled ratchet. 



♦ e 



"IT 



without feedback 

with feedback 

X 7 

with feedback 



The number of switches 



FIG. 8: Numerical tests of the Jarzynski equality for the flash- 
ing ratchet and a generalized Jarzynski equality (|89p for the 
feedback-controlled ratchet. 



VII. CONCLUSIONS 

In this paper, we have studied the effects of measure- 
ments and feedback control on nonequilibrium thermo- 
dynamic systems. In particular, we have generalized 
nonequilibrium equalities to the systems that are sub- 
ject to feedback control. Our formulations and results 
are applicable to a broad class of classical nonequilibrium 
systems. 

In Sec. II, we reviewed stochastic thermodynamics, by 
focusing on the nonequilibrium equalities. In Sec. Ill, 
we formulated measurements on nonequilibrium systems, 
and defined mutual information Ic by (|3ip for multi- 
measurements. In Sec. IV, we formulated feedback con- 
trol on nonequilibrium systems. We discussed the prop- 
erties of the joint probability (|47|) . which is well-defined 
due to causality. We introduced the mutual information 
Ic by (|55p . which is not equivalent to / in the presence of 
feedback control. In fact, Ic describes the correlation be- 
tween the system and the outcomes, which characterizes 
the effective information obtained by the measurements. 
We have also shown that the detailed fluctuation theorem 
(|55)) holds in the presence of feedback control. 



Section V constitutes the main results of this paper. 
We derived two types of generalizations of the nonequi- 
librium equalities. In Sec. V A, we derived a generalized 
detailed fiuctuation theorem (j65p which involves the mu- 
tual information. Based on Eq. (|65p . we derived the gen- 
eralizations of the integral fiuctuation theorem (|66P . the 
Jarzynski equality ([7T|) . the second laws ([57)1 ([7^ ([751) . 
the fiuctuation-dissipation theorem (|70p . and the KPB 
equality (j68p that all involve the mutual information. In 
Sec. V B, we derived the renormalized detailed fluctua- 
tion theorem ([55]) , and derived the generalizations of the 
integral fiuctuation theorem (j87p . the Jarzynski equality 
(|89| , the second law (iQlT) , the fluctuation-dissipation the- 
orem (|92P . and the KPB equality (|96p . We have shown 
that mutual information Ic, rather than /, plays the cru- 
cial role to formulate the nonequilibrium equalities under 
feedback control. These results are the generalizations of 
the fundamental equalities in nonequilibrium statistical 
mechanics to feedback-controlled processes, and lead to 
the generalized second law of thermodynamics with feed- 
back control, which gives the minimal energy cost that is 
needed for the feedback control. 

In Sec. VI, we discussed simple examples to explicitly 
show that our results in Sec. V can be applied to typical 
situations. In Sec. VI A, we discussed the Szilard en- 
gine with measurement errors that achieves the equality 
of the generalized second law of thermodynamics (j72p or 
(|73p . This is an important model to quantitatively illus- 
trate that the mutual information can be converted to the 
work. We also confirmed the two generalized Jarzynski 
equalities ([7T|) ([89]) in the generalized Szilard engine. In 
Chap. VI B, we considered a feedback-controlled ratchet 
and confirmed a generalized Jarzynski equality (j89p . 

All of our formulations and results are consistent with 
the original nonequilibrium equalities and the second law 
of thermodynamics, and our results serve as the funda- 
mental principle of nonequilibrium thermodynamics of 
feedback control. We note that, in our results such as 
Eq. ([S5|) . the thermodynamic quantities and the infor- 
mation contents are treated on an equal footing. There- 
fore, our theory may be regarded as the nonequilibrium 
version of "information thermodynamics" [94l . |96| , which 
serves as the fundamental theory of nonequilibrium in- 
formation heat engines. 



Appendix A: Physical Meaning of the Entropy 
Production 



In this appendix, we discuss the physical meanings of 
the entropy production a in the following two typical se- 
tups to clarify the typical situations to which our results 
apply. 

Isothermal processes. We assume that there is a sin- 
gle heat bath at temperature T = (/cb/?)"^, and that the 
initial distributions of both forward and backward exper- 
iments are in the canonical distributions. We stress that 
we do not assume that the final distributions of both the 
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forward and backward experiments are in the canonical 
distributions: the final distribution of the forward (back- 
ward) experiments does not necessarily equal the initial 
distribution of the backward (forward) experiments. Let 
H{x, A) be the Hamiltonian of the system with the time 
symmetry H{x,X) — H(x* , X*). The canonical distribu- 
tion with parameter A is given by 

P_[:,|A]:=e'5(^(^)-^(-^^)), (Al) 

where 

F(A) := -keT In J dxe^'^-^^^'^) (A2) 

is the Helmholtz free energy. In this situation, the en- 
tropy production reduces to 

a[XN]^l3{W[XN]-AF), (A3) 

where 

W[Xn] := H{xN,XRn) - H{xo,Xi^t) - Q[Xn] (A4) 

is the work performed on the system from the external 
parameter, and AF := F(Afin)— i^(Aint) is the free-energy 
difference. In this case, Eq. leads to the Jarzynski 
equality (ITOl) , and the second law ([T5|) reduces to inequal- 
ity (Uni). 

Transition between arbitrary nonequilibrium states: 
We assume that there are several heat baths, and that 
we can control the strength of interaction between the 
system and the baths through A. In other words, we 
can attach or detach the system from the baths by con- 
trolling A; for example, we can attach an adiabatic wall 
to the system. We set an arbitrary initial distribution 
Po[2^o] for the forward experiments. On the other hand, 
the initial state of the backward experiments is assumed 
to be taken as -P(|[a:J] := Pn[xn], where Pn[xn] is the 
final distribution of the forward experiments. Although 
this choice of the backward initial state is artificial and is 



difhcult to be experimentally realized except for special 
cases, this backward initial state is a theoretically useful 
tool to derive a version of the second law of thermody- 
namics as follows. In this case, the entropy production 
is given by 

a[XN] = -\nPN[xN]+\nPo[xo] - ^AQ4^iv], (A5) 

i 

and its ensemble average leads to 

(a) = 5Ar-5o-^ft(Q,), (A6) 

where 

Sn ■■= - J Pn [Xn] In P„ [Xn]dXn (A7) 

is the Shannon entropy at time i„ . By introducing nota- 
tion AS* := Sn — Sq, the second law leads to 

AS>Y,MQ'^)- (A8) 

i 
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